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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  will  be  a  “paper  only”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resolution,  with  the  attendant  restrictions  on  travel  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
colleagues  and  the  opportunity  for  all  our  researchers  to  present  their  work.  However,  we 
intend  to  simulate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  published  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduled  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Naval  Postgraduate  School  (NPS)  continues  at  a  normal  pace.  Since  the  ARP’s 
founding  in  2003,  over  1,200  original  research  reports  have  been  added  to  the  acquisition 
body  of  knowledge.  We  continue  to  add  to  that  library,  located  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SHIPS 
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•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 

•  Office  of  Procurement  and  Assistance  Management  Headquarters, 
Department  of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test,  & 
Evaluation 

•  Program  Executive  Officer,  Tactical  Aircraft 

•  Director,  Office  of  Small  Business  Programs,  Department  of  the  Navy 

•  Director,  Office  of  Acquisition  Resources  and  Analysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 

•  Director  of  Open  Architecture,  DASN  (RDT&E) 

•  Program  Executive  Officer,  Littoral  Combat  Ships 

James  B.  Greene  Jr.  Keith  F.  Snider,  PhD 

Rear  Admiral,  U.S.  Navy  (Ret.)  Associate  Professor 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  ii  - 


Acquisition  Management 


Naval  Ship  Maintenance:  An  Analysis  of  the  Dutch  Shipbuilding  Industry  Using  the 
Knowledge  Value  Added,  Systems  Dynamics,  and  Integrated  Risk  Management 
Methodologies 

David  N.  Ford,  Thomas  J.  Housel,  and  Johnathan  C.  Mun 
Naval  Postgraduate  School 

Time  as  an  Independent  Variable:  A  Tool  to  Drive  Cost  Out  of  and  Efficiency  Into 
Major  Acquisition  Programs 

J.  David  Patterson 

National  Defense  Business  Institute,  University  of  Tennessee 

The  Impact  of  Globalization  on  the  U.S.  Defense  Industry 

Jacques  S.  Gansler  and  William  Lucyshyn 
University  of  Maryland 

Bottleneck  Analysis  on  the  DoD  Pre-Milestone  B  Acquisition  Processes 

Danielle  Worger  and  Teresa  Wu,  Arizona  State  University 
Eugene  Rex  Jalao,  Arizona  State  University  and  University  of  the  Philippines 
Christopher  Auger,  Lars  Baldus,  Brian  Yoshimoto,  J.  Robert  Wirthlin,  and  John 
Colombi,  The  Air  Force  Institute  of  Technology 

Software  Acquisition  Patterns  of  Failure  and  How  to  Recognize  Them 

Lisa  Brownsword,  Cecilia  Albert,  Patrick  Place,  and  David  Carney 
Carnegie  Mellon  University 

Fewer  Mistakes  on  the  First  Day:  Architectural  Strategies  and  Their  Impacts  on 
Acquisition  Outcomes 

Linda  McCabe  and  Anthony  Wicht 
Massachusetts  Institute  of  Technology 

The  Joint  Program  Dilemma:  Analyzing  the  Pervasive  Role  That  Social  Dilemmas  Play 
in  Undermining  Acquisition  Success 

Andrew  P.  Moore,  William  E.  Novak,  Julie  B.  Cohen,  Jay  D.  Marchetti,  and  Matthew 
L.  Collins 

Software  Engineering  Institute,  Carnegie  Mellon  University 

Acquisition  Risks  in  a  World  of  Joint  Capabilities:  A  Study  of  Interdependency 
Complexity 


- 1  - 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


Mary  Maureen  Brown 

University  of  North  Carolina  Charlotte 

Leveraging  Structural  Characteristics  of  Interdependent  Networks  to  Model  Non- 
Linear  Cascading  Risks 

Anita  Raja,  Mohammad  Rashedul  Hasan,  and  Shalini  Rajanna 
University  of  North  Carolina  at  Charlotte 

Ansaf  Salleb-Aoussi,  Columbia  University,  Center  for  Computational  Learning 
Systems 

Lexical  Link  Analysis  Application:  Improving  Web  Service  to  Acquisition  Visibility 
Portal 

Ying  Zhao,  Shelley  Gallup,  and  Douglas  MacKinnon 
Naval  Postgraduate  School 

Capturing  Creative  Program  Management  Best  Practices 

Brandon  Keller  and  J.  Robert  Wirthlin 
Air  Force  Institute  of  Technology 

The  RITE  Approach  to  Agile  Acquisition 

Timothy  Boyce,  Iva  Sherman,  and  Nicholas  Roussel 
Space  and  Naval  Warfare  Systems  Center  Pacific 

Challenge-Based  Acquisition:  Stimulating  Innovative  Solutions  Faster  and  Cheaper 
by  Asking  the  Right  Questions 

Richard  Weatherly,  Virginia  Wydler,  Matthew  D.  Way,  Scott  Anderson,  and  Michael 
Arendt 

MITRE  Corporation 

Defense  Acquisition  and  the  Case  of  the  Joint  Capabilities  Technology  Demonstration 
Office:  Ad  Hoc  Problem  Solving  as  a  Mechanism  for  Adaptive  Change 

Kathryn  Aten  and  John  T.  Dillard 
Naval  Postgraduate  School 

A  Comparative  Assessment  of  the  Navy’s  Future  Naval  Capabilities  (FNC)  Process 
and  Joint  Staff  Capability  Gap  Assessment  Process  as  Related  to  Pacific  Command’s 
(PACOM)  Integrated  Priority  List  Submission 

Jaime  Frittman,  Sibel  McGee,  and  John  Yuhas,  Analytic  Services,  Inc. 

Ansaf  Salleb-Aoussi,  Columbia  University 

Enabling  Design  for  Affordability:  An  Epoch-Era  Analysis  Approach 

Michael  A.  Schaffner,  Marcus  Wu  Shihong,  Adam  M.  Ross,  and  Donna  H.  Rhodes 
Massachusetts  Institute  of  Technology 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-2- 


Measuring  Dynamic  Knowledge  and  Performance  at  the  Tactical  Edges  of 
Organizations:  Assessing  Acquisition  Workforce  Quality 

Mark  E.  Nissen 

Naval  Postgraduate  School 

Outcome-Focused  Market  Intelligence:  Extracting  Better  Value  and  Effectiveness 
From  Strategic  Sourcing 

Timothy  G.  Hawkins,  Naval  Postgraduate  School 

Michael  E.  Knipper,  771  Enterprise  Sourcing  Squadron  USAF 

Timothy  S.  Reed,  Beyond  Optimal  Strategic  Solutions 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-3- 


Measuring  Dynamic  Knowledge  and  Performance  at  the 
Tactical  Edges  of  Organizations:  Assessing  Acquisition 

Workforce  Quality 


Mark  E.  Nissen — Nissen  is  a  professor  of  information  science  and  management  at  the  Naval 
Postgraduate  School.  His  research  focuses  on  dynamic  knowing  and  organizing.  He  views  work, 
technology,  organization,  and  people  as  an  integrated  design  problem,  and  he  has  concentrated  for 
some  time  on  the  dynamics  of  tacit  and  explicit  knowledge  flows,  looking  in  particular  at  (re)designing 
organizations  that  balance  stability  with  maneuverability.  Nissen’s  150+  publications  span  information 
systems,  project  management,  organization  studies,  knowledge  management,  and  related  fields. 
Before  his  doctoral  work  at  the  University  of  Southern  California,  he  acquired  over  a  dozen  years' 
management  experience  in  the  aerospace  and  electronics  industries,  [mnissen@nps.edu] 

Abstract 

The  efficacy  of  defense  acquisition  is  highly  dependent  on  acquisition  workforce  (AWF) 
quality,  but  assessing  such  quality  remains  a  major  challenge,  particularly  given  the 
knowledge-intensive  and  dynamic  nature  of  acquisition  organizations  and  processes.  Hence, 
it  is  difficult  to  gauge — much  less  predict — the  impact  of  leadership  interventions  in  terms  of 
policy,  process,  regulation,  organization,  education,  training,  or  like  approaches.  Building 
upon  the  development  and  application  of  Knowledge  Flow  Theory  over  the  past  couple  of 
decades,  we  have  developed  a  state-of-the-art  approach  that  enables  us  to  analyze, 
visualize,  and  measure  dynamic  knowledge  and  performance.  The  main  idea  is  to  apply  this 
approach  inwardly  to  measure  the  dynamic  knowledge  and  performance  of  acquisition 
processes  (e.g.,  within  contracting  and  project  management  organizations),  but  we  also  look 
outwardly  (e.g.,  at  warfare  processes  at  the  tactical  edges  of  military  combat  organizations)  to 
conceptualize  an  operational  proxy  for  acquisition  workforce  quality:  end  customer 
performance.  This  proxy  offers  its  best  potential  to  complement,  not  replace,  other  metrics  in 
use,  development,  and  conceptualization  today,  but  it  arguably  concentrates  on  one  of  the 
most  important  AWF  quality  determinants:  how  acquired  systems  affect  operational 
performance. 

Introduction 

Acquisition  is  big  business.  The  DoD  alone  routinely  executes  12-figure  budgets  for 
research,  development,  procurement,  and  support  of  weapon  systems  and  other  military 
products  and  services  (Dillard  &  Nissen,  2005).  Acquisition  is  also  a  knowledge-intensive 
business.  In  addition  to  myriad  laws  governing  federal  acquisition  in  the  U.S.,  a  plethora  of 
rules  and  regulations  specify — often  in  great  detail — how  to  accomplish  the  planning,  review, 
execution,  and  oversight  of  defense  acquisition  programs,  large  and  small,  sole-source  and 
competitive,  military  and  commercial  (Dillard,  2003). 

As  a  result  in  part — and  due  to  high  complexity,  multiple  stakeholders,  goal 
incongruence,  open  process  execution,  and  large  pecuniary  rewards  for  some  participants — 
acquisition  has  been  a  problematic  business  too.  Seemingly  every  decade,  acquisition 
problems  must  be  addressed  by  another  Blue  Ribbon  panel  and  reformed  yet  again.  The 
Better  Buying  Power  Initiatives  (BBPI),  as  a  recent  instance,  mandate  efficiency  and 
productivity  improvements  in  five  acquisition  business  areas:  (1)  affordability  and  cost 
growth,  (2)  productivity  and  innovation  in  industry,  (3)  competition,  (4)  tradecraft  in  services 
acquisition,  and  (5)  non-productive  processes  and  bureaucracy  (Office  of  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  [OUSD(AT&L)],  2010). 
These  initiatives  focus  principally  on  incentives  for  and  interactions  with  contractors.  The 
Defense  Acquisition  Workforce  Improvement  Act  (DAWIA),  as  another  instance,  was  signed 
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into  law  in  1990  and  emphasizes  the  education,  training,  and  certification  of  people  in  the 
acquisition  workforce  (AWF).  Of  course,  the  two  leadership  interventions  are  related:  people 
in  the  AWF  need  to  know  how  to  effect  the  kinds  of  efficiency  and  productivity  improvements 
mandated  via  the  BBPI. 

These  characteristics  of  acquisition  emphasize  the  criticality  of  quality  in  the  AWF 
itself:  With  so  much  at  stake,  and  in  such  a  knowledge-intensive  environment,  a  high-quality 
workforce  is  essential  to  competent  and  professional  acquisition  performance. 

These  characteristics  also  elucidate  the  central  role  played  by  people  and 
organizations  in  the  AWF.  People  must  be  knowledgeable  and  work  effectively — both  in 
terms  of  their  own  professional  acquisition  activities  and  with  many  others  in  acquisition  and 
customer  organizations — in  order  to  accomplish  key  objectives  and  ensure  timely, 
affordable,  and  responsive  delivery  of  products  and  services  to  fighting  and  support  units,  at 
home  and  abroad.  Indeed,  we  understand  well  how  the  efficacy  of  defense  acquisition  is 
inextricably  dependent  on  workforce  quality.  Hence,  leadership  interventions  along  these 
lines  appear  to  be  highly  appropriate  and  on  target. 

Assessing  the  impact  of  interventions  such  as  these  is  a  challenge,  however 
(Assistant  Secretary  of  the  Navy  for  Research,  Development,  and  Acquisition  [ASN(RDA)], 
2011a,  2011b).  It  is  unclear  whether  the  relatively  recent  BBPI,  for  instance,  has  had 
sufficient  time  to  produce  measurable  impact.  Even  after  two  decades  of  the  DAWIA,  as 
another  instance,  efficacy  remains  challenging  to  assess,  for  many  extant  measures  (e.g., 
number  of  Defense  Acquisition  University  graduates,  procurement  lead  times,  program  cost 
growth)  fail  to  account  for  critical  aspects  of  the  AWF  and  important  impacts  on  acquisition 
performance.  Indeed,  it  is  difficult  to  gauge — much  less  predict — the  impact  of  any 
leadership  interventions  along  these  lines  (e.g.,  how  much  better  the  AWF  has  become,  or 
even  if  it  is  improving  over  time).  Hence,  the  impact  of  any  particular  leadership  intervention 
is  left  largely  to  anecdote  and  optimism.  To  help  trim  acquisition  budgets  and  guide 
leadership,  an  improvement  in  assessing  leadership  initiatives  and  interventions  is  needed. 

Because  acquisition  is  a  knowledge-intensive  endeavor  (Snider  &  Nissen,  2003),  the 
knowledge  stocks  of  people  comprising  the  AWF  represent  likely  indicators  of  quality  (e.g., 
education  levels,  training  courses,  years  of  experience,  certification  levels).  However,  such 
indicators  are  relatively  static,  pertaining  to  levels  of  knowledge  that  change  comparatively 
slowly  (Nissen,  2006a).  In  contrast,  acquisition  laws,  rules,  and  regulations  are  revised 
frequently,  and  acquisition  knowledge  can  change  abruptly  and  render  obsolete  even  huge 
stocks  over  time.  Indeed,  this  dynamic  acquisition  environment  requires  members  of  the 
AWF  to  sustain  career-long  learning  and  knowledge  development  just  to  remain  proficient 
as  acquisition  professionals.  Thus,  as  indicators  of  AWF  quality,  static  knowledge  stocks 
appear  to  be  out  of  phase  with  the  highly  dynamic  nature  of  the  acquisition  environment. 

Moreover,  acquisition  organizations  experience  persistent  flux  (Snider  &  Nissen, 
2003).  We  understand  well  that  no  two  acquisition  projects,  programs,  organizations, 
customers,  or  requirements  are  completely  alike.  Hence,  even  well-educated  and  well- 
trained  people,  with  appropriate  certification  levels  and  years  or  decades  of  acquisition 
experience,  must  continually  learn  afresh  and  expand  their  knowledge  further  with  each  new 
assignment.  Likewise,  it  is  clear  that  most  acquisition  organizations  form  and  reform  with 
new  people  (e.g.,  via  personnel  transfer,  turnover,  retirement,  promotion)  continuously  and 
that  end  customer  needs  shift  perennially  (especially  at  the  tactical  edges  of  warfare 
organizations).  Due  to  such  discontinuous  membership  (Ibrahim  &  Nissen,  2007),  even 
these  educated,  trained,  certified,  and  experienced  people  must  learn  repeatedly  to  trust 
and  work  effectively  with  many  others — each  time  someone  new  joins  or  leaves  a  particular 
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acquisition  organization,  and  each  time  a  novel  product,  service,  or  customer  is  involved. 
Thus,  dynamic  knowledge  also  appears  to  be  an  important  AWF  quality  indicator. 

Further,  the  pace  of  change  in  both  information  technologies  and  military  operations 
causes  this  importance  of  dynamic  knowledge  to  apply  in  particular  where  information 
systems  (IS)  are  acquired  to  support  people  at  the  tactical  edges  of  warfare  organizations. 
Not  only  must  acquisition  personnel  be  competent  in  their  professions — including  the 
acquisition  and  maintenance  of  new  acquisition  knowledge  and  skills — and  continually  learn 
afresh  amidst  constant  organizational  flux,  but  they  must  also  keep  pace  with  incessant 
technological  change  and  satisfy  customers’  dynamic  needs.  Even  highly  competent 
professionals  executing  internal  acquisition  processes  perfectly  can  fail  to  satisfy  end 
customers’  materiel  or  service  needs.  This  presents  a  huge  challenge  in  terms  of  assessing 
AWF  quality. 

Building  on  the  development  and  application  of  Knowledge  Flow  Theory  (KFT)  over 
the  past  couple  of  decades  (see  Nissen,  2006b),  we  have  developed  a  state-of-the-art 
approach  that  enables  us  to  analyze,  visualize,  and  measure  dynamic  knowledge  and 
performance.  This  measurement-based  approach  offers  potential  to  overcome  the 
limitations  of  static  measures,  as  previously  summarized,  by  focusing  inwardly  on  the 
dynamics  of  knowledge  important  to  professional  and  effective  acquisition  performance.  The 
main  idea  is  to  measure  the  dynamic  knowledge  and  performance  of  acquisition  processes 
(e.g.,  within  contracting  and  project  management  organizations).  This  would  represent  a 
substantial  step  forward  in  terms  of  acquisition  research. 

Further,  leveraging  complementary  research  in  command  and  control  (C2;  Nissen  & 
Gallup,  2012),  we  see  potential  to  use  this  same  measurement-based  approach  to  also  look 
outwardly  at  the  dynamics  of  knowledge  important  to  professional  and  effective  warfare 
performance.  Although  the  specific  kinds  of  knowledge  required  for  effective  warfare  will 
clearly  differ  from  those  essential  for  proficient  acquisition,  the  approach  is  similar.  The  main 
idea  is  to  measure  the  dynamic  knowledge  and  performance  of  warfare  processes  (e.g.,  at 
the  tactical  edges  of  military  combat  organizations).  This  would  represent  a  substantial  step 
forward  in  terms  of  C2  research. 

Moreover,  we  seek  to  link  these  inward  and  outward  focusing  approaches  to 
conceptualize  an  operational  proxy  for  AWF  quality:  end  customer  performance.  In  addition 
to  measuring  the  dynamic  knowledge  and  performance  of  key  people  and  organizations 
associated  with  IS  acquisition,  for  instance,  we  wish  to  assess  AWF  quality  by  also 
measuring  the  dynamic  knowledge  and  performance  of  primary  beneficiaries  of  such 
systems  acquisition:  end  customers  operating  at  the  tactical  edges  of  warfare  organizations. 
This  proxy  offers  its  best  potential  to  complement,  not  replace,  other  metrics  in 
conceptualization,  development,  and  use  today,  but  it  arguably  concentrates  on  one  of  the 
most  important  AWF  quality  determinants:  how  acquired  systems  affect  operational 
performance.  Two  fundamental  research  questions  follow  accordingly: 

1 .  How  can  dynamic  knowledge  and  performance  metrics  be  applied  to  assess 
acquisition  workforce  quality? 

2.  How  can  dynamic  knowledge  and  performance  metrics  be  extended  to  the 
tactical  edges  of  warfare  organizations? 

Building  heavily  on  the  exploratory  study  reported  by  Nissen  (2012b),  we  summarize 
fast-changing  IS  acquisition  from  the  perspective  of  warfare  at  the  tactical  edge,  and  we 
discuss  dynamic  knowledge  and  performance  measures  to  both  complement  and  contrast 
with  extant,  engineering-oriented  metrics  used  to  specify  and  assess  most  acquired  systems 
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today.  We  begin  with  a  summary  of  KFT  and  measurement  and  then  follow  with  the 
research  method  guiding  the  study.  Results  follow  and  suggest  considerable  promise, 
particularly  where  acquisition  personnel  and  organizations  can  learn  and  track  how 
changing  system  characteristics  correspond  with  operational  performance  at  the  tactical 
edges  of  warfare  organizations  over  time. 

Background 

The  dynamic  nature  of  knowledge  indicates  that  both  stocks  and  flows  are  important 
(Dierickx  &  Cool,  1989).  Knowledge  stocks  have  been  comparatively  straightforward  to 
measure  historically;  metrics  pertaining  to  education  levels,  training  courses,  years  of 
experience,  certifications,  and  like  knowledge-oriented  factors  are  employed  broadly. 
Alternatively,  knowledge  flows  have  been  comparatively  much  more  difficult  to  assess; 
metrics  pertaining  to  dynamic  knowledge — particularly  at  the  group  and  organization 
levels — are  more  elusive.  The  development  and  application  of  KFT  (see  Nissen,  2006b) 
over  the  past  couple  of  decades  has  augmented  the  set  of  tools  and  techniques  available  to 
analyze,  visualize,  and  measure  dynamic  knowledge  and  performance  in  the  organization. 

KFT  is  founded  on  a  set  of  30  principles  that  characterize  dynamic  knowledge.  Such 
principles  are  actionable  and  empirical,  and  they  support  the  diagnosis  of  workflow  and 
knowledge-flow  process  pathologies,  visualization  of  improvement  interventions,  and 
measurement  of  dynamic  knowledge  and  performance  gains  (Nissen,  2006a).  Dynamic 
knowledge  is  delineated  via  five-dimensional  (5D)  vector  space.  Knowledge-flow  vectors 
carry  measurements  and  elucidate  diagnostic  inferences  pertaining  to  the  people, 
processes,  and  organizations  associated  with  knowledge  work.  Figure  1  illustrates  the  idea. 

Briefly,  the  vertical  axis,  “Explicitness,”  characterizes  the  nature  of  knowledge  along 
a  tacit-explicit  continuum.  Tacit  knowledge  implies  understanding  and  know-how/why,  and  it 
is  associated  most  closely  with  the  experiences  of  people  (e.g.,  stemming  from  job 
assignments,  mentoring,  and  teamwork)  and  routines  of  organizations  (including  culture, 
process,  and  ritual).  Explicit  knowledge  implies  awareness  and  know-who/what/where/when, 
and  it  is  associated  most  closely  with  artifacts  (e.g.,  documents,  formulae,  software). 
Generally,  the  more  tacit  the  knowledge,  the  greater  its  appropriability  and  potential  impact 
on  positive  performance  becomes  (Saviotti,  1998).  One  can  measure  knowledge 
explicitness  using  ordinal,  interval,  or  ratio  scales. 


Figure  1.  5D  Knowledge  Flow  Diagram 
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The  horizontal  axis,  “Reach,”  characterizes  how  broadly  knowledge  is  known  and 
shared  in  an  organization.  Here  we  operationalize  reach  in  terms  of  the  number  of  people  in 
an  organization  who  have  access  to  and  can  employ  any  particular  chunk  of  knowledge,  but 
we  could  view  reach  in  terms  of  organizational  levels  instead  (e.g.,  individual,  group, 
organization,  interorganization).  Generally,  the  broader  the  reach  of  knowledge,  the  greater 
its  amplification  and  potential  impact  on  positive  performance  becomes  (Nonaka,  1994). 
Measurements  can  be  made  using  ordinal,  interval,  or  ratio  scales. 

The  axis  “Life  cycle”  characterizes  what  is  being  done  with  a  particular  chunk  of 
knowledge  at  some  specific  point  in  time.  Here  we  include  three  activities:  (1)  some 
individual  in  the  organization  learns  or  creates  new  knowledge;  (2)  he  or  she  shares  existing 
knowledge  with  or  transfers  it  to  other  people  in  the  organization;  and  (3)  one  or  more 
people  in  the  organization  use  or  apply  existing  knowledge  to  accomplish  work.  Generally, 
knowledge  does  not  become  useful  until  it  is  used  or  applied  (Pfeffer  &  Sutton,  1999). 
Measurements  can  be  made  using  categorical  or  ordinal  scales. 

Because  visualization  beyond  three  dimensions  is  difficult,  we  represent  the 
dimension  “Flow  time”  in  terms  of  the  thickness  of  lines  used  to  delineate  vectors.  As  shown 
in  the  key  to  the  right  of  Figure  1,  relatively  thin  lines  are  used  to  delineate  short  and  fast 
knowledge  flows,  whereas  comparatively  thick  lines  represent  knowledge  that  takes  a  long 
time  and  flows  slowly.  Generally,  the  more  quickly  that  knowledge  flows  (e.g.,  across 
people,  organizations,  places,  times),  the  greater  its  potential  impact  on  positive 
performance  becomes  (Nissen,  2002).  Measurements  can  be  made  using  ordinal,  interval, 
or  ratio  scales. 

The  dimension  “Power”  is  represented  similarly  in  terms  of  line  style  used  to 
delineate  knowledge-flow  vectors.  Knowledge  that  flows  with  relatively  low  power — this 
corresponds  with  relatively  low  performance  levels  of  organizational  activities  enabled  by  the 
knowledge — is  delineated  through  orange,  dotted  lines,  whereas  knowledge  flows  exhibiting 
high  power — and  hence  enabling  high  performance — are  delineated  via  purple,  solid  lines. 
Measurements  can  be  made  using  ordinal,  interval,  or  ratio  scales. 

Integrating  these  five  dimensions  graphically  and  analytically  generates  a  5D  vector 
space  to  examine  dynamic  knowledge.  Such  5D  space  and  examination  schemes  are 
completely  general:  they  can  be  applied  to  any  dynamic  knowledge  in  any  organizational 
domain  (e.g.,  acquisition,  C2,  software  engineering). 

As  an  example  of  use  and  application,  consider  Figure  2,  which  illustrates  an 
important  knowledge  flow  desired  by  the  organization.  Point  A  represents  one  individual  in 
the  organization  who  learns  something  new  (to  that  organization)  or  creates  entirely  new 
knowledge.  In  terms  of  the  5D  space,  this  represents  tacit  knowledge  that  is  created  by  an 
individual  (i.e.,  one  person);  hence,  its  position  at  the  bottom-back  corner  of  the  diagram. 
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Figure  2.  Knowledge  Creation  and  Application  Needs 


In  the  acquisition  domain,  for  instance,  consider  that  such  new  knowledge  could 
pertain  to  a  technique  for  reducing  the  acquisition  time  for  an  important  IS  needed  in  the 
field.  Because  information  technology  (IT)  advances  so  quickly — outpacing  the  ability  of 
many  acquisition  organizations  to  develop  and  field  systems  responsively — the  organization 
views  this  new  knowledge  created  at  Point  A  as  important,  and  it  would  like  to  see  such 
knowledge  shared  with  and  applied  by  all  100  people  in  that  organization  who  work  with  IT. 

Such  application  by  100  people  in  the  organization  is  represented  by  Point  B.  The 
thin,  purple,  solid  vector  connecting  Points  A  and  B  represents  the  desired  knowledge  flow: 
the  organization  wishes  for  such  knowledge  to  flow  quickly  and  with  high  power  (e.g., 
enabling  all  100  people  at  Point  B  to  work,  within  one  day,  at  the  same  performance  level  as 
the  innovative  individual  at  Point  A).  This  represents  a  5D  knowledge  flow  vector.  A  question 
mark  in  the  figure  next  to  the  vector  indicates  that  such  a  fast,  powerful  knowledge  flow  is 
desired  by  the  organization,  but  it  is  unclear  which,  if  any,  organizational  process  can  enable 
it. 


This  leads  to  Figure  3,  which  depicts  a  ridge,  or  obstruction,  that  prevents  knowledge 
from  flowing  quickly  and  powerfully  from  Points  A  to  B  as  desired  by  the  organization. 
Practically,  the  organization  lacks  a  process  for  such  quick  and  powerful  knowledge  to  flow 
directly  as  delineated  in  Figure  2.  Indeed,  most  organizations  do  lack  such  a  process 
(Nissen,  2006a).  Some  other  approach  to  sharing  and  applying  the  important  IT  acquisition 
knowledge  is  required. 
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Figure  4  delineates  two  alternate,  archetypical  knowledge  flows  corresponding  to 
processes  that  are  within  this  organization’s  capabilities.  (We  say  archetypical  because  most 
organizations  employ  these  classic  processes  routinely,  and  because  they  present  a  vivid 
contrast  in  terms  of  how  dynamic  knowledge  flows.)  One  knowledge  flow  is  depicted  in 
terms  of  a  relatively  fast  (i.e.,  thin  lines)  but  low-power  (i.e.,  orange,  dotted  lines)  vector 
series;  this  first  flow  is  associated  with  explicit  knowledge  and  utilizes  one  or  more  IS  for 
knowledge  articulation  and  distribution  in  explicit  form.  The  other  is  delineated  via  a 
comparatively  slow  (i.e.,  thick  lines)  but  high-power  (i.e.,  purple,  solid  lines)  vector;  this 
second  flow  is  associated  with  tacit  knowledge  and  utilizes  one  or  more  human-centered 
approaches  to  knowledge  sharing  (e.g.,  group  interaction,  mentoring,  personnel  transfer). 


Figure  4.  Alternate  Archetypical  Knowledge  Flows 

In  some  greater  detail,  the  first  knowledge  flow  consists  of  three  vectors.  The  first 
vector  is  represented  by  a  vertical  line  arising  from  Point  A.  This  vector  depicts  the  individual 
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at  Point  A  articulating  his  or  her  new,  tacit  knowledge  via  an  IS  so  that  it  can  be  shared 
electronically.  Such  articulation  (e.g.,  consider  writing  a  procedure,  developing  a  training 
course,  posting  to  an  intranet  or  social  networking  site)  tends  to  be  somewhat  time- 
consuming,  hence  the  relatively  thick  line.  Articulating  knowledge  in  explicit  form  also  tends 
to  dilute  the  knowledge  in  terms  of  power.  Reading  a  book,  for  instance,  about  how  to 
accomplish  important  acquisition  tasks  (e.g.,  contract  negotiation,  risk  assessment, 
balancing  program  cost  and  schedule  with  performance)  is  not  the  same  as  having  direct 
personal  experience  accomplishing  those  tasks,  hence  the  orange,  dotted  line. 

Once  articulated  in  explicit  form,  however — particularly  via  IS — the  knowledge  can  be 
shared  very  broadly  (e.g.,  organization-wide)  and  very  quickly  (e.g.,  within  seconds),  albeit 
with  diluted  power,  hence  the  thin,  orange,  dotted  line  at  the  top  of  the  diagram.  Indeed,  one 
could  consider  this  broad  and  fast  flow  as  additive  to  the  organization’s  express  acquisition 
body  of  knowledge  (BOK),  which  we  note  at  the  top-right  of  Figure  4.  Such  an  explicit  BOK 
can  then  be  accessed  quickly  and  applied  in  turn  by  all  100  people  in  the  organization.  This 
articulated,  explicit  knowledge  remains  relatively  diluted  and  less  powerful,  nonetheless,  so 
application  at  Point  B  would  not  support  the  same  performance  level  as  at  Point  A,  hence 
the  thin,  orange,  dotted  line  descending  down  to  Point  B. 

Alternatively,  the  second  knowledge  flow  consists  of  a  single  vector,  although  it 
curves  and  bends  through  the  tacit  knowledge  plane  at  the  bottom  of  Figure  4.  This  vector 
depicts  the  individual  at  Point  A  applying  his  or  her  new,  tacit  knowledge  and  then  sharing  it 
with  some  number  of  other  people  (say,  10  people,  as  illustrated  in  Figure  4)  through  one  or 
more  techniques,  such  as  extended  group  interaction,  mentoring,  or  personnel  transfer  to 
work  directly  with  different  coworkers  across  the  organization. 

Once  each  of  these  10  people  has  learned  the  new,  tacit  knowledge,  then  all  of  them 
can  continue  the  process  and  share  it  using  similar  techniques  (e.g.,  group  interaction, 
mentoring,  or  personnel  transfer)  with  others.  Through  such  a  process,  100  people  (i.e. ,  10 
people  each  sharing  with  another  10  people)  can  learn  this  new,  tacit  knowledge  to  the 
extent  necessary  for  powerful  application  at  Point  B.  This  knowledge  flow  is  depicted  by  a 
thick  vector  to  indicate  that  it  occurs  comparatively  slowly,  but  such  vector  is  also  delineated 
by  a  purple,  solid  line  to  show  that  the  corresponding  knowledge  has  high  power  and 
enables  knowledge-based  action  at  the  same  performance  level  as  the  individual  who 
created  it  at  Point  A. 

The  key  is  that  one  can  measure  these  five  dimensions  of  knowledge — whether  via 
explicit  or  tacit  flows — and  relate  them  to  the  corresponding  knowledge-based  process 
performance  by  people  in  the  organization.  Indeed,  by  correlating  such  dynamic  knowledge 
measures  with  performance  metrics,  one  can  develop  a  model  capable  of  analyzing, 
visualizing,  and  even  predicting  process  performance  based  on  knowledge  flow  patterns. 

Of  course,  many  diverse  combinations  of  these  archetypical  knowledge  flows  are 
possible  too,  yet  most  knowledge  flows  are  likely  to  reflect  some  aspects  of  these  two 
dynamic  patterns  (Nissen,  2006b).  Through  empirical  analysis  and  calibration  of  specific 
knowledge  flowing  through  any  particular  organization  in  the  field,  one  can  correlate  5D 
dynamic  knowledge  flows  with  work  performance,  resulting  in  a  model  capable  of 
measurement  and  prediction.  Through  this  technique,  we  are  working  to  assess  AWF  quality 
in  terms  of  dynamic  knowledge  flows. 

Research  Method 

The  first  research  question,  articulated  previously,  includes  a  “how”  interrogative  and 
suggests  that  a  qualitative  method  may  be  most  appropriate  to  investigate  it  (Yin,  1994). 
Despite  the  generality  of  KFT  and  the  5D  space  described  in  the  previous  section,  applying 
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the  corresponding  analytic,  visualization,  and  measurement  techniques  to  assess  AWF 
quality  requires  acquisition  domain  knowledge  in  general  and  process-specific 
understanding  in  particular.  We  need  to  study  one  or  more  specific  acquisition  processes  in 
detail  in  order  to  apply  the  techniques  and  assess  workforce  quality.  The  case  study  method 
is  highly  appropriate  for  an  investigation  along  these  lines  (Benbasat,  Goldstein,  &  Mead, 
1987;  Eisenhardt  &  Graebner,  2007;  Yin,  1994),  and  we  conduct  just  such  a  case  study  in 
parallel  with  the  investigation  reported  here. 

The  second  research  question,  stated  previously,  also  involves  a  “how”  interrogative, 
and  it  likewise  suggests  a  qualitative  method.  However,  this  second  question  calls  for  an 
extension  of  dynamic  knowledge  and  performance  measurement  out  to  the  tactical  edges  of 
warfare  organizations  and  hence  is  much  more  exploratory  from  an  acquisition  perspective. 
Because  we  seek  an  operational  proxy  for  AWF  quality,  we  investigate  dynamic  knowledge 
and  performance  through  explicit  examination  of  three  warfare  organizations  and  processes 
that  are  far  removed  from  core  acquisition. 

One  organization  operates  within  a  U.S.  Navy  fleet  and  has  units  deploying 
rhythmically  to  war  zones  and  other  areas  overseas.  A  second  organization  operates  within 
a  Navy  systems  command  but  concentrates  on  ensuring  the  readiness  of  this  same  fleet. 
The  third  organization  permeates  functionally  throughout  naval  operations  and  is 
responsible  for  information  dominance.  By  interacting  with  knowledgeable  representatives 
from  each  of  these  three  organizations — and  it  is  very  important  to  note  that  these  are 
warriors  and  other  operational  personnel,  not  acquisition  professionals — we  gain 
considerable  insight  into  the  key  knowledge  dynamics  associated  with  warfare  at  the  tactical 
edges. 

Further,  by  triangulating  between  these  three  organizations,  we  identify  a  critical, 
knowledge-intensive  process  that  can  be  represented  with  sufficient  fidelity  and  granularity 
to  suggest  feasible  application  of  our  dynamic  knowledge  and  performance  measures.  The 
process  has  the  somewhat  unwieldy  name  Tasking,  Collection,  Processing,  Exploitation, 
and  Dissemination,  to  which  we  refer  simply  by  its  acronym  TCPED.  In  the  results  that 
follow,  we  delineate  the  TCPED  process  and  seek  to  apply  our  dynamic  knowledge  and 
performance  measures  to  it.  We  then  attempt  to  interpret  such  application  and  to  elucidate 
insight  into  assessing  AWF  quality  via  proxy. 

Results 

Results  from  this  exploratory  investigation  center  on  delineating  the  TCPED  process, 
elaborating  an  insightful  subprocess  in  detail,  and  applying  our  dynamic  knowledge  and 
performance  measures  to  it.  We  discuss  these  in  turn  and  then  focus  on  elucidating  insight 
into  AWF  quality. 

TCPED 

TCPED  does  not  represent  a  new  operational  process  per  se,  but  with  the  U.S. 
Navy’s  relatively  recent  creation  of  its  Information  Dominance  Corps,  it  has  attracted 
considerable  attention  as  a  critical  complement  to  the  find,  fix,  target,  and  track  (F2T2) 
process  associated  broadly  with  combat  operations.  The  key  F2T2  issue  remains 
“knowledge — finding  the  targets”  (Keeter,  2004),  and  as  a  knowledge-intensive  process, 
TCPED  addresses  this  issue  directly,  and  hence  represents  a  promising  target  of  study. 

Given  the  knowledge-intensive  nature  of  TCPED,  its  execution  is  enabled 
fundamentally  by  IT,  and  IS  are  acquired  routinely  with  the  goal  of  enhancing  warfare 
efficacy.  This  nature  provides  an  excellent  link  back  to  our  fundamental  research  question 
and  interest  in  the  AWF.  From  the  operational  perspective  of  TCPED  participants  at  the 
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tactical  edges  of  organizations,  IS  acquired  and  fielded  to  enhance  warfare  efficacy  should 
accomplish  just  that:  enhance  warfare  efficacy.  Further,  such  efficacy  enhancement  should 
be  measurable. 

The  problem  is,  it  is  difficult  to  understand — much  less  measure — how  well  any 
particular  warfare  process  is  working,  which  of  many  different  organizational  arrangements 
are  best  across  diverse  missions,  or  how  well  various  IS  enhance  or  impede  the  process. 
Indeed,  when  seeking  to  acquire  new  IT  and  like  technologies  to  enhance  warfare  efficacy, 
system  implementation  can  make  the  operational  processes  worse  in  the  battle  space,  and 
it  is  increasingly  common  for  different  acquired  systems  to  fail  in  terms  of  interoperating 
(Nissen  &  Gallup,  2012). 

Indeed,  modern  warfare  efficacy  requires  a  combination  of  people  and  technologies 
to  enable  warriors  to  leverage  local  knowledge  and  seize  emergent  opportunities  to  achieve 
commanders’  intent  across  distributed  organizations.  This  requirement  highlights  further  the 
critical  role  played  by  TCPED,  which  seeks  to  enable  commanders  and  warriors  at  the 
tactical  edges  to  put  dynamic  knowledge  into  effective  action,  with  or  without  IS  in 
development  or  in  the  field. 

Additionally,  unlike  many  stable,  mature,  and  well-understood  warfare  processes, 
TCPED  remains  in  a  constant  state  of  analysis,  refinement,  and  development.  Hence,  it 
represents  a  rapidly  moving  target  for  IT  development,  and  engineering-oriented  metrics 
used  to  evaluate  most  IS  fail  to  address  how  dynamic  knowledge  translates  into  effective  (or 
ineffective)  action.  Moreover,  with  current  analytical  models  and  metrics,  it  remains  unclear 
how  to  assess  whether  any  particular  refinement  in  the  warfare  process,  new  IS 
implementation,  or  like  change  will  lead  to  increased  TCPED  efficacy  or  whether 
performance  will  degrade  instead.  This  lack  of  clarity  illuminates  a  capacious  gap  between 
the  efficiency  of  IT  acquisition  and  the  warfare  efficacy  of  IS  employment  at  the  tactical 
edge. 

Given  the  dynamic  nature  of  the  TCPED  process,  as  characterized  previously,  we 
bound  the  scope  of  this  exploratory  project  by  concentrating  on  a  particularly  important  and 
knowledge-centric  subprocess:  exploitation.  Such  bounding  enables  us  to  examine,  within  a 
single  exploratory  study,  the  feasibility  of  our  approach  to  measuring  the  dynamic 
knowledge  and  performance  of  this  operational  process  performed  at  the  tactical  edges  of 
naval  organizations.  Follow-on  researchers  can  then  extend  these  promising  results  via 
subsequent  studies  through  the  process  as  a  whole  and,  in  turn,  to  other  warfare  processes 
seeking  to  benefit  from  IT  acquisition. 

TCPED  Exploitation 

Figure  5  delineates  the  principal  tasks  comprising  TCPED  exploitation.  In  this  figure, 
process  activities  are  depicted  as  rectangular  boxes  connected  to  one  another  via  arrows  to 
delineate  the  process  workflow.  Each  process  activity  is  situated  within  a  horizontal  region 
(referred  to  widely  as  swim  lanes)  that  depicts  the  responsibility  of  a  particular  organizational 
group  to  accomplish  it.  For  several  instances,  the  leftmost  process  activities — “Correlate, 
Fuse  Multi-lnt  Info”  “Operations  Environment  Impact”  “Evaluate  Adversary” 

“Develop  Adversary  COA” — are  shown  connected  together  as  responsibilities  of  the 
“Assessor”  group;  the  “Develop  Adversary  COA”  activity  interrelates  with  “Watch  Analyst 
Coordination,”  the  latter  of  which  is  shown  as  the  responsibility  of  the  Joint  Intelligence 
Operations  Center  (JIOC),  and  which  interrelates  in  turn  with  the  Joint  Operations  Center 
(JOC)  activity  “Watch  Coordination.” 
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Figure  5.  TCPED  Exploitation  Process  Flow 

Other  instances  pertain  to  “Assess  Near-Term  Ops  Impact,”  the  output  of  which 
activity  provides  important  knowledge  and  information  to  Operations  (“J3”);  “Daily  Update 
Information”  and  “Propose  New  Focus  Areas,”  the  output  of  which  activities  provide 
important  knowledge  and  information  to  intelligence  management;  and  “Determine 
Emerging  l&W”  and  “Dissem  l&W,”  both  of  which  activities  are  performed  by  and  are  the 
responsibility  of  the  assessor  as  well.  We  omit  graphical  depiction  or  discussion  of  the  other 
TCPED  exploitation  activities  because  our  intent  is  not  to  be  exhaustive  here,  and  these 
should  suffice  for  our  present  purposes. 

In  particular,  discussions  with  the  knowledgeable  people  interviewed  through  this 
research  indicate  that  the  tasks  labeled  “Evaluate  Adversary,”  “Develop  Adversary  COA,” 
and  “Assess  Near-Term  Ops  Impact”  are  especially  important  and  require  considerable  tacit 
knowledge.  Recall  that  tacit  knowledge,  as  powerful  as  it  is,  tends  to  flow  relatively  slowly 
and  narrowly  through  organizations.  This  makes  it  particularly  challenging  to  support  via  IT, 
and  it  provides  an  excellent  focus  for  our  exploration.  Indeed,  the  people  performing  these 
activities  must  develop  substantial,  tacit  knowledge  pertaining  to  adversaries’  capabilities, 
likely  actions,  and  their  consequences  in  terms  of  friendly  forces  and  operations.  Such  tasks 
also  clearly  require  relevant  and  timely  information,  but  knowledge  of  the  adversary  is  key 
here,  and  the  effectiveness  of  these  tasks  can  contribute  greatly  to — or,  if  ineffective,  impair 
instead — commanders’  decision-making  and  warriors’  actions  on  the  tactical  edge. 

By  focusing  on  how  dynamic  knowledge  flows  through  warfare  process  activities 
such  as  these,  and  especially  by  linking  the  activities  to  knowledge-based  actions  enabled 
at  the  tactical  edge,  we  can  examine  how  well  knowledge  is  flowing  and  supporting  tactical 
action.  Specifically,  by  integrating  the  organizations,  personnel,  and  activities  included  in  the 
exploitation  process  diagrammed  in  Figure  5  with  key  dimensions  from  KFT,  we  seek  to 
identify  critical  paths  in  the  process  where  knowledge  is  flowing  well  and  appropriately,  as 
well  as  identifying  blocked  paths  where  it  is  not,  and  we  strive  to  use  our  dynamic 
knowledge  and  performance  metrics  to  help  overcome  any  disconnects  between  IT 
acquisition  and  warfare  efficacy. 
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Dynamic  Knowledge  Flows 

Through  detailed  analysis,  we  can  delineate  the  principal  knowledge  flows  enabling 
TCPED  exploitation.  Taking  Develop  Adversary  COA  as  an  express  example,  the  people 
performing  this  activity  rely  fundamentally  on  experience-based  tacit  knowledge  (e.g., 
military  tactics,  adversary  capabilities,  organizational  vulnerabilities).  Although  formal 
training  courses,  professional  educational  programs,  and  like  approaches  contribute  to 
these  knowledge  flows,  such  knowledge  is  accumulated  principally  through  direct 
experience  (i.e.,  on-the-job  training  [OJT]),  often  over  many  years  or  even  decades. 


Figure  6.  Military  Tactics  Knowledge  Flows 


Figure  6  delineates  how  military  tactics  knowledge,  for  instance,  accumulates 
through  cyclic  iteration  between  applying  one’s  existing  tacit  knowledge  (labeled  “Tactics  K 
application”  in  the  figure)  and  learning  from  the  resulting  experience  (labeled  “Tactics  K 
creation”  in  the  figure).  We  locate  this  cyclic  knowledge  flow  vector  at  the  individual  level  of 
reach,  indicating  that  the  Develop  Adversary  COA  activity  is  conducted  in  this  case  by  a 
single  individual.  Were  multiple  people  to  engage  jointly  in  assessments  such  as  this,  we 
would  simply  relocate  the  corresponding  knowledge  flows  to  the  group  level,  with  the  same 
basic  pattern  persisting. 

Consistent  with  our  previous  discussion,  one  can  observe  from  Figure  6  how  the 
vector  for  knowledge  application  is  relatively  thin,  denoting  that  the  flow  is  correspondingly 
fast;  yet  this  vector  is  delineated  via  a  purple,  solid  arrow,  denoting  that  the  flow  reflects 
powerful,  tacit  knowledge.  That  is,  once  tacit  knowledge  has  been  acquired  over  time,  it  can 
be  applied  relatively  quickly.  In  partial  contrast,  the  complementary  vector  for  knowledge 
creation  is  comparably  thick,  denoting  that  the  knowledge  acquisition  flow  is  relatively  slow; 
yet  this  vector  is  also  delineated  via  a  purple,  solid  arrow,  similarly  denoting  that  the  flow 
reflects  powerful,  tacit  knowledge. 

Continuing  with  the  Develop  Adversary  COA  example,  the  people  performing  this 
activity  also  rely  on  a  situated  understanding  of  the  organization’s  current  mission- 
environment  context,  the  adversary  evaluation  synthesized  in  the  preceding  exploitation 
process  step,  and  contemporaneous  knowledge  regarding  both  current  and  future 
operations  being  conducted  and  planned,  respectively,  by  the  organization.  Knowledge 
flowing  to  enable  these  process  activities  follows  somewhat  different  patterns  than  those 
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activities  pertaining  to  military  tactics.  In  particular,  these  latter  knowledge  flows  involve 
interactions  across  different  organizational  groups,  and  they  involve  both  tacit  and  explicit 
knowledge. 

For  instance,  Figure  7  delineates  three  knowledge  flows  associated  with  tacit 
knowledge  sharing  and  intergroup  accumulation.  The  leftmost  cyclic  vector  (labeled 
“Individual  K  accumulation”)  is  comparable  to  that  discussed  previously  in  Figure  6,  except 
that  instead  of  military  tactics  knowledge,  it  pertains  to  the  latter  knowledge  flows  (e.g., 
associated  with  current  mission-environment,  adversary  evaluation,  and  current  and  future 
operations).  We  continue  to  focus  on  individual  knowledge  accumulated  by  a  single 
person — in  this  case,  within  the  assessor  group — but  notice  that  we  include  a  similar  cyclic 
vector  located  at  the  intergroup  level. 


Figure  7.  Tacit  Knowledge  Sharing  and  Intergroup  Accumulation 

This  latter  vector  (labeled  “Intergroup  K  accumulation”)  reflects  tacit  knowledge 
accumulating  across  different  organizational  groups;  multiple  individuals  from  a  variety  of 
groups  work  and  learn  from  their  experiences  together.  The  intergroup  vector  follows  the 
same  cyclic  pattern  as  that  seen  with  individual  OJT,  only  at  a  higher  organizational  level. 

As  with  individual  knowledge  accumulation,  this  intergroup  accumulation  is  delineated  by  a 
cyclic,  purple,  solid  vector  reflecting  knowledge  application  and  creation  occurring  at  two 
different  rates:  quickly  and  slowly,  respectively. 

A  third  vector  (labeled  “Tacit  K  sharing”)  links  the  other  two.  Such  tacit  knowledge 
sharing  reflects  individuals — who  accumulate  knowledge  (especially  via  OJT)  within  their 
separate  groups — sharing  knowledge  with  people  in  other  groups  through  conversation, 
dialogue,  face-to-face  (F2F)  interaction,  and  like  means.  The  two-headed  arrow  included 
with  this  sharing  vector  depicts  knowledge  flowing  bi-directionally:  individuals  share 
knowledge  across  groups  in  the  organization,  and  they  also  learn  through  this  knowledge 
process. 

As  with  the  two  cyclic  vectors  delineated  and  discussed  previously,  knowledge  flows 
corresponding  to  such  tacit  sharing  are  depicted  with  a  purple,  solid  vector  to  designate 
powerful  tacit  knowledge,  and  the  vector  is  depicted  with  a  relatively  thick  line  to  indicate 
that  tacit  knowledge  flows  across  organizational  groups  tend  to  accumulate  relatively  slowly. 
However,  we  depict  this  sharing  vector  with  a  line  that  exhibits  intermediate  thickness;  that 
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is,  the  vector  is  thicker  than  the  application  vector  lines — suggesting  that  tacit  knowledge 
application  flows  across  groups  (e.g.,  in  a  matter  of  days,  weeks,  and  months)  more  slowly 
than  via  individual  application  (e.g.,  in  a  matter  of  minutes,  hours,  and  days) — but  thinner 
than  the  creation  vectors — suggesting  that  such  cross-group  knowledge  can  flow  more 
quickly  than  can  individual  accumulation  of  experience-based  tacit  knowledge  (e.g.,  in  a 
matter  of  months,  years,  and  decades). 


Figure  8.  Explicit  Organizational  Knowledge  Sharing 

As  another  instance,  Figure  8  delineates  alternate  knowledge  flows  associated  with 
explicit  organizational  knowledge  sharing.  The  leftmost  cyclic  vector  (labeled  “Individual  K 
accumulation”)  is  identical  to  that  discussed  previously  in  Figure  7  (e.g.,  cyclic,  purple,  solid, 
powerful,  tacit).  We  continue  to  focus  on  individual  knowledge  accumulated  by  a  single 
person — in  this  case,  within  the  assessor  group — but  notice  that  we  include  a  three-segment 
flow  (labeled  “Explicit  K  sharing”)  to  depict  knowledge  being  shared  organization-wide  in 
explicit  form. 

This  three-segment  flow  begins  with  a  vertical  vector  rising  up  out  of  the  tacit  plane, 
as  an  individual  (i.e.,  in  the  assessor  group)  articulates  his  or  her  tacit  knowledge  into 
explicit  form  (e.g.,  via  textual  reports,  graphical  sketches,  digital  images).  This  articulation 
can  be  a  time-consuming  process;  hence,  the  corresponding  knowledge  flow  vector  is 
depicted  by  a  relatively  thick  line.  In  addition,  we  understand  that  such  articulated,  explicit 
knowledge  does  not  reflect  the  same  power  level  as  the  tacit  knowledge  used  for  its 
creation;  hence,  the  corresponding  knowledge  flow  vector  is  depicted  by  an  orange,  dotted 
line. 


The  second  vector  comprising  this  three  segment  flow  begins  where  the  first  vector 
terminates.  Once  articulated  in  explicit  form,  such  knowledge  can  be  stored,  replicated,  and 
disseminated  quickly  and  broadly  via  one  or  more  IS  (e.g.,  intranet  document  repositories, 
online  sharing  tools,  common  operational  displays).  This  second  vector  in  the  segment  is 
delineated  by  a  thin  line  to  denote  fast  knowledge  flows,  but  the  line  remains  orange  and 
dotted  to  depict  its  diluted  power.  The  third  vector  in  the  segment  is  also  depicted  by  a  thin, 
orange,  dotted  vector,  which  represents  this  same,  diluted,  explicit  knowledge  flowing  via  IS 
quickly  and  broadly  across  the  organization. 
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Dynamic  Knowledge  and  Performance  Measurement 

Through  detailed  analysis,  we  identify  and  operationalize  three  KFT  metrics  that 
appear  to  be  particularly  insightful  for  our  present  purposes:  knowledge  reach  (i.e. ,  how 
many  people  in  the  organization  share  specific  chunks  of  knowledge),  knowledge  flow  time 
(i.e.,  how  long  it  takes  chunks  of  tacit  and  explicit  knowledge  to  flow  from  where  and  when 
they  are  to  where  and  when  they  are  needed),  and  knowledge  power  (i.e.,  the  performance 
level  of  knowledge-enabled  work).  Continuing  with  Develop  Adversary  COA  as  an  express 
example,  we  can  begin  to  quantify  the  key  knowledge  flows  delineated  previously. 


Table  1.  ROOM  Knowledge  Flow  Measurement 


Knowledge  Flow 

Reach 

Flow  Time 

Power 

Individual  K  Accumulation 

1 

Years 

Very  High 

Intergroup  K  Accumulation 

10 

Months 

High 

Tacit  K  Sharing 

10 

Days 

High 

Explicit  K  Sharing 

100 

Hours 

Diluted 

For  instance,  Table  1  summarizes  rough  order  of  magnitude  (ROOM),  3D  estimates 
for  each  of  the  four  knowledge  flows  delineated  and  discussed  previously  with  respect  to  the 
Develop  Adversary  COA  activity  within  TCPED  exploitation.  In  this  table,  we  approximate 
knowledge  flow  measurements  only  to  an  order  of  magnitude,  but  we  begin  to  illustrate  the 
use  and  utility  of  the  approach,  and  we  outline  a  method  for  obtaining  higher  fidelity 
measurements  in  practice. 

In  the  first  column  of  the  table,  we  list  each  of  the  four  knowledge  flows  discussed 
previously;  and  in  the  other  three  columns,  we  summarize  ROOM  estimates  for  knowledge 
reach,  flow  time,  and  power.  Looking  first  at  individual  knowledge  accumulation,  the  reach  is 
listed  as  1;  this  reflects  our  previous  discussion  of  knowledge  being  accumulated  iteratively 
at  the  individual  level,  hence  unitary  reach.  In  the  table,  flow  time  is  listed  in  order  of 
magnitude  as  “years”  for  comparison  with  the  other  knowledge  flows;  this  reflects  our 
discussion  about  how  deep,  experience-based  tacit  knowledge  (e.g.,  pertaining  to  military 
tactics)  can  require  years  or  decades  to  accumulate.  Power  is  listed  likewise  in  order  of 
magnitude  as  “very  high”  for  similar  comparison  with  the  other  knowledge  flows;  this 
estimate  is  somewhat  definitional,  but  it  reflects  that  experience-based  tacit  knowledge  does 
not  suffer  from  power  dilution,  and  it  is  meant  to  reflect  the  considerable  power  of  tacit 
knowledge  accumulated  over  long  periods  of  time  and  through  abundant  experience. 

Looking  next  at  intergroup  knowledge  accumulation,  rough  estimates  for  this 
knowledge  flow  indicate  that  10  people  can  be  reached  by  it;  this  is  an  order  of  magnitude 
larger  than  that  shown  for  individual  knowledge  accumulation,  and  it  reflects  knowledge 
flowing  to  multiple  people  across  organizational  groups.  The  flow  time  estimated  for 
intergroup  knowledge  flows  is  summarized  as  “months,”  which  is  an  order  of  magnitude 
faster  than  that  for  individual  knowledge  accumulation;  this  reflects  the  comparatively  lower 
level  of  deep  knowledge  associated  with  intergroup  knowledge  and  work  flows,  as  people 
across  groups  interact  principally  via  their  present  assignments — which,  in  this  naval 
context,  generally  span  less  than  a  year.  As  discussed  previously,  the  power  level  is  listed 
simply  as  “high”  to  reflect  that  intergroup  tacit  knowledge  (e.g.,  people  learning  to  work  well 
together  across  groups)  does  not  suffer  from  power  dilution,  but  it  also  reflects  that  the 
power  level  is  not  comparable  to  that  associated  with  deep,  experience-based  knowledge 
accumulated  over  years  of  individual  experience  (e.g.,  pertaining  to  military  tactics). 
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Estimates  for  the  third  knowledge  flow  (i.e.,  tacit  knowledge  sharing)  are  similar  in 
terms  of  reach  (10),  but  they  reflect  more  than  another  order  of  magnitude  reduction  in  flow 
time  (i.e.,  “days”);  this  corresponds  to  the  principle  that  knowledge  sharing  can  be 
accomplished  more  quickly  than  the  associated  knowledge  accumulation  (Nissen,  2006b). 
The  (“high”)  power  level  matches  that  for  intergroup  accumulation  mentioned  previously  and 
for  the  same  reasons. 

In  considerable  contrast,  the  flows  associated  with  the  fourth  knowledge  flow  (i.e., 
explicit  knowledge  sharing)  are  quantitatively  very  different.  We  estimate  the  reach  at  100  in 
the  table,  but  the  knowledge  flows  are  constrained  only  by  the  reach  of  the  network 
infrastructure;  hence,  this  figure  could  be  many  orders  of  magnitude  larger  (e.g.,  consider  a 
report,  through  which  everyone  in  a  100,000  person  organization  has  access  to  the  same 
explicit  knowledge).  The  estimate  for  flow  time  is  similar  in  that  we  list  it  as  “hours”  (e.g., 
principally  to  account  for  the  time  required  to  articulate  knowledge  in  explicit  form),  whereas 
once  made  explicit,  such  knowledge  can  be  shared  in  seconds. 

Moreover,  the  power  level  (“diluted”)  for  this  explicit  knowledge  flow  is  qualitatively 
different  from  that  for  its  tacit  counterparts;  this  is  also  somewhat  definitional,  but  it  indicates 
that  most  people  reading  written  documents,  for  example,  will  not  be  expected  to  perform 
knowledge-based  activities  at  the  same  level  as  the  people  writing  those  documents. 

System  Assessment 

The  remaining  measurement  of  knowledge  power  is  linked  directly  to  performance  of 
the  work  activities  enabled  by  such  knowledge.  In  the  case  of  Develop  Adversary  COA,  to 
continue  our  previous  example,  we  could  approach  such  measurement  via  multiple 
operationalizations.  For  several  instances,  we  could  track  how  much  time  is  required  to 
develop  a  set  of  adversary  COAs  sufficiently  well  for  inclusion  in  a  morning  flag  brief  (i.e., 
appropriate  for  presentation  to  a  flag  officer);  using  the  same  flag  brief  criterion,  we  could 
count  how  many  sufficiently  credible  adversary  COAs  are  developed  within  a  set  time  frame 
(e.g.,  one  day,  week,  or  month);  we  could  ask  the  flag  officer  and  staff  in  question  (including 
the  Chief  of  Staff  and  other  directly  reporting  officers)  to  evaluate  the  quality  of  each 
adversary  COA  presented  (based  on  criteria  of  importance  to  them);  or  we  could  pursue  the 
development  of  other,  likewise  understandable  and  relevant  performance  measures.  Any 
such  performance  measure  can  serve  as  a  quantitative  (and  possibly  multidimensional) 
proxy  for  knowledge  power. 

With  one  or  more  such  measures  in  hand,  we  could  then  establish  a  baseline — 
comprised  of  quantitative  measurements  for  reach,  flow  time,  and  knowledge 
power/performance — for  the  organization  as  it  operates  as  usual.  To  evaluate  some 
particular  IS,  we  could  simply  compare  this  baseline  with  measurements  taken  as  the 
organization  uses  the  IS  under  controlled,  or  at  least  comparable,  conditions.  For  instance, 
say  that  we  wish  to  test  a  prototype  IS  designed  to  improve  tacit  knowledge  sharing  through 
introduction  of  social  media  techniques;  we  could  measure  the  knowledge  flows  both  with 
and  without  such  IS  to  assess  its  impacts. 

Specifically,  using  one  or  more  proxy  measures  as  suggested  previously  (e.g.,  time 
required  to  develop  a  set  of  adversary  COAs  for  a  flag  brief,  how  many  adversary  COAs  are 
developed,  flag  officer  quality  evaluation,  others),  we  could  conduct  an  experiment  in  the 
laboratory  or  in  the  “field”  (e.g.,  on  deployed  ships  at  sea)  and  measure  knowledge  and 
performance  directly.  As  an  experiment  to  compare  performance  with  and  without  the 
prototype  IS,  for  instance,  we  would  ideally  like  to  see  the  same  people,  performing  the 
same  tasks,  in  the  same  environments  and  settings,  at  the  same  times  of  day,  seasons  of 
year,  weather  conditions,  sea  states,  and  other  factors  to  isolate  use  of  that  IS  as  the  only 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-295  - 


difference.  In  other  words,  one  set  of  dynamic  knowledge  measurements  would  be  taken  for 
performance  in  the  baseline  situation;  a  second  set  of  measurements  would  be  taken  for 
performance  with  a  prototype  IS;  and,  ideally  (e.g.,  with  good  experiment  design  and 
techniques),  the  difference  would  represent  solely  the  effect  of  that  IS. 

With  these  measurements  in  hand,  the  difference  in  task  performance  becomes  an 
operational  measure  of  IS  efficacy;  that  is,  if  the  only  difference  between  experiment  cases 
is  whether  the  prototype  IS  used  or  not,  and  task  performance  is  measurably  better  or  worse 
in  one  case  or  the  other,  then  we  have  a  knowledge-based  assessment  of  how  well  such  IS 
improves  (or  worsens)  work  performance  at  the  tactical  edge  of  the  organization  (e.g., 
TCPED  exploitation).  Moreover,  in  addition  to  using  traditional,  engineering-oriented 
performance  measures  (e.g.,  bandwidth,  technical  reliability,  memory),  this  assessment  can 
be  employed  to  evaluate  the  IS  operationally — and  under  controlled  conditions — not  just 
technically.  The  potential  is  huge. 

Further,  given  sufficient  experience  with  conducting  experiments  along  these  lines, 
this  approach  can  even  be  used  to  specify  new  IT  and  other  systems  to  be  acquired;  that  is, 
in  conjunction  with  using  only  engineering  measures  of  IS  performance,  for  instance,  the 
acquisition  organization  can  specify  improvement  in  operational  task  performance  as  a  key 
criterion  for  evaluation.  This  way,  acquisition  personnel  can  conduct  efficient  system 
acquisitions,  and  warriors  on  the  tactical  edges  of  organizations  can  use  systems  that 
improve  their  work  performance.  We  bridge  the  gap  between  acquirer  and  warrior,  and 
everybody  wins. 

Illustrative  Example 

In  this  section,  we  include  an  illustrative  example  of  application  to  a  hypothetical  IT 
system  competition.  We  use  only  representative  values  for  illustration  here,  but  the 
approach  and  associated  techniques  can  be  applied  directly  to  system  competitions  in  the 
field.  For  continuity,  we  continue  with  the  Develop  Adversary  COA  task  discussed 
previously,  and  we  build  upon  the  rough  knowledge  flows  and  measurements  reported 
previously. 

Table  2.  Baseline  Knowledge  Flow  Measurement 


Knowledge  Flow 

Reach 

Flow  Time 

Power 

X-Power 

Tacit  K  Sharing 

10 

20  Hours 

95% 

9.5 

Explicit  K  Sharing 

100 

2  Hours 

5% 

5.0 

Table  2  recapitulates  the  most  relevant  measurements  reported  in  Table  1  for  what 
we  term  the  baseline,  representing  the  Develop  Adversary  COA  task  as  it  is  performed 
today  (i.e.,  sans  new  IS);  that  is,  the  baseline  measurements  are  used  for  comparison  with 
this  same  task  performed  with  the  support  of  two  competing  IS  prototypes:  (1)  a  social 
media  application  designed  to  improve  tacit  knowledge  sharing,  versus  (2)  a  document 
collaboration  application  designed  to  improve  explicit  knowledge  sharing. 

Notice  in  Table  2  that  we  limit  our  summary  to  the  pair  of  knowledge  flows 
associated  directly  with  the  alternate  IS:  tacit  knowledge  sharing  (addressed  by  IS-1)  and 
explicit  knowledge  sharing  (addressed  by  IS-2).  Recall,  from  our  discussion  above,  that  the 
knowledge  flow  corresponding  to  tacit  knowledge  sharing  reflects  individuals — who 
accumulate  knowledge  (especially  via  OJT)  within  their  separate  groups — sharing 
knowledge  with  people  in  other  groups  through  conversation,  dialogue,  F2F  interaction,  and 
like  means.  The  central  idea  of  IS-1  is  to  enable  such  knowledge  sharing  remotely;  that  is, 
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the  IS  intends  to  enable  and  promote  tacit  knowledge  sharing  without  the  need  for  (as 
much)  F2F  interaction. 

Recall,  further  from  our  discussion  above,  that  the  knowledge  flow  corresponding  to 
explicit  knowledge  sharing  reflects  organizational  artifacts  (e.g.,  textual  reports,  graphical 
sketches,  digital  images,  and  like  media)  that  are  stored,  replicated,  and  disseminated 
quickly  and  broadly  via  intranet  document  repositories,  online  sharing  applications,  common 
operational  displays,  and  like  tools.  The  central  idea  of  IS-2  is  to  enable  recipients  of 
assessor  reports  (e.g.,  in  the  JIOC  and  JOC  groups)  to  interact  with  assessors  during  report 
development;  that  is,  the  IS  intends  to  enhance  and  accelerate  explicit  knowledge  sharing 
by  providing  recipients  with  access  to  assessor  draft  reports  and  to  enable  communication 
before  finished  reports  are  released  officially. 

Notice  also  that  we  replace  the  ROOM  estimates  from  Table  1  with  quantitative 
values.  For  instance,  the  “days”  flow  time  estimate  from  above  for  the  tacit  knowledge 
sharing  flow  reads  as  “20  hours”  in  Table  2.  Based  on  observation  and  discussion,  roughly 
20  hours  are  required  for  key  tacit  knowledge  to  complete  its  flows.  Further,  the  “high”  power 
estimate  from  above  reads  as  “95%”  here.  As  such,  10  different  people  outside  the  assessor 
group  (e.g.,  in  the  JIOC  and  JOC)  are  able  to  explain  the  details  of  each  adversary  COA 
from  memory  with  95%  accuracy  on  average;  the  other  way  to  look  at  this  is  that  19  of  20 
people  can  explain  the  details  with  100%  accuracy. 

Similarly,  the  “hours”  flow  time  estimate  from  above  for  the  explicit  knowledge 
sharing  flow  reads  as  “2  hours”  here.  This  indicates  that  roughly  two  hours  are  required  for  a 
high-quality  and  credible  adversary  COA  to  be  articulated,  shared  with,  and  understood  by 
recipients.  Further,  the  “diluted”  power  estimate  from  above  reads  as  5%  here.  As  such,  100 
different  people  outside  the  assessor  group  (e.g.,  in  the  JIOC  and  JOC)  are  able  to  explain 
the  details  of  each  adversary  COA  from  memory  with  5%  accuracy  on  average;  the  other 
way  to  look  at  this  is  that  five  of  100  people  can  explain  the  details  with  100%  accuracy. 

Notice,  finally,  that  we  include  a  fifth  column  in  Table  2  (labeled  “X-Power”)  to 
represent  the  induced  dimension  extended  knowledge  power.  Extended  knowledge  power  is 
calculated  as  the  product  of  knowledge  reach  and  power  levels;  it  reflects  the  combined 
distribution  and  efficacy  of  knowledge  flows.  For  instance,  the  extended  knowledge  power 
for  the  tacit  knowledge  sharing  flow  is  shown  in  Table  2  as  9.5  (i.e.,  reach  of  10  [times] 
power  of  .95  =  x-power  of  9.5),  whereas  the  value  calculated  for  explicit  knowledge  sharing 
flow  is  shown  as  5.0  (i.e.,  reach  of  100  [times]  power  of  .05  =  x-power  of  5.0). 

This  respective  induction  and  quantification  of  the  extended  knowledge  power 
dimension  and  measure  provide  us  with  a  technique  for  comparing  the  efficacy  of  tacit  and 
explicit  knowledge  flows  directly,  despite  the  significant  differences  between  their  dynamic 
characteristics  and  behaviors  (e.g.,  quick,  broad,  diluted  explicit  flows  versus  slow,  narrow, 
powerful  tacit  flows).  Clearly,  higher  values  are  preferred  over  lower  ones,  but  organizations 
face  trade-offs  regarding  whether  to  emphasize  explicit  or  tacit  knowledge  flows. 

Table  3.  Information  Systems  Supported  Knowledge  Flow  Measurement 


Knowledge  Flow 

Reach 

Flow  Time 

Power 

X-Power 

Tacit  K  Sharing  (IS-1) 

20 

20  Hours 

75% 

15.0 

Explicit  K  Sharing  (IS-2) 

20 

2  Hours 

10% 

2.0 

For  further  illustration,  Table  3  summarizes  these  same  knowledge  flow 
measurements — for  the  same  people,  organizations,  tasks,  and  time  frames — after  the 
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prototype  IS  have  been  implemented  and  trained  with.  This  point  is  important;  one  cannot 
expect  a  new  IS  to  be  used  effectively  and  productively  before  its  users  have  been  trained 
adequately.  (It  is  humorous,  nonetheless,  how  often  one  sees  comparisons  made  without 
adequate  training,  particularly  in  field  experiments.) 

In  the  case  of  tacit  knowledge  sharing  supported  by  IS-1,  say  that  the  social  media 
application  enables  twice  as  many  people  to  participate  in  the  conversations  (i.e.,  reach 
extends  to  20)  within  the  same  20-hour  time  frame  (e.g.,  by  obviating  the  need  for 
collocation),  but  the  power  level  decreases  to  75%  (e.g.,  due  to  losses  via  mobile  social 
media  applications).  Despite  the  drop  in  power,  the  extended  reach  would  more  than  make 
up  for  the  loss,  because  of  the  extended  power  increase  to  15.0.  Alternatively,  in  the  case  of 
explicit  knowledge  sharing  supported  by  IS-2,  say  that  the  document-sharing  application 
reduces  to  20  the  number  of  people  who  can  participate  effectively  within  the  same  two-hour 
time  frame  (e.g.,  due  to  interference  by  multiple  people  interacting  with  the  same 
documents),  yet  the  power  level  of  those  who  do  participate  increases  to  10%  (e.g., 
stemming  from  increased  textual  interaction  across  organizational  groups).  Despite  the 
increase  in  power,  the  reduced  reach  would  more  than  offset  the  gain  because  of  the 
extended  power  decrease  to  2.0. 


Table  4.  Comparative  Knowledge  Flow  Measurement 


Knowledge  Flow 

Baseline 

IS  Enabled 

Difference 

Difference 

(X-Power) 

(X-Power) 

(X-Power) 

(Percentage) 

Tacit  K  Sharing 

9.5 

15.0 

+  5.5 

+  58% 

Explicit  K  Sharing 

5.0 

2.0 

-3.0 

-  60% 

In  Table  4,  we  summarize  the  comparative  results  via  four  measurements.  First,  the 
Baseline  X-Power  contrast  between  the  tacit  and  explicit  knowledge  sharing  processes 
reflects  our  result  from  Table  2  (i.e.,  9.5  versus  5.0,  respectively).  Second,  the  IS  Enabled  X- 
Power  contrast  between  these  same  processes  reflects  similarly  our  result  from  Table  3 
(i.e.,  15.0  versus  2.0,  respectively).  Third,  the  Difference  X-Power  contrast  measures  the 
effect  of  incorporating  the  two  IS.  For  instance,  using  IS-1  to  support  tacit  knowledge 
sharing  increases  extended  knowledge  power  by  5.5  (i.e.,  15.0  -  9.5  =  +5.5)  for  a  58%  gain. 
In  contrast,  using  IS-2  to  support  explicit  knowledge  sharing  decreases  extended  power  by 
3.0  (i.e.,  2.0  -  5.0  =  -3.0)  for  a  60%  loss. 

Recall  that  the  knowledge  power  measurement  relates  directly  to  organizational 
performance  at  the  tactical  edge,  on  the  Develop  Adversary  COA  task  in  this  illustrative 
case.  In  addition  to  providing  an  objective  and  quantitative  approach  to  assessing  the 
potential  value  (or  harm)  of  an  IS  of  interest,  the  technique  described  in  this  report  also 
suggests  a  way  to  specify  performance  requirements  for  candidate  IS  of  interest. 

Consider,  for  instance,  if — in  addition  to  whatever  engineering  specifications  are 
desirable  or  customary — the  specification  read  along  the  lines  of,  “the  IS  must  demonstrate 
at  least  a  25%  increase  in  X-Power  measured  during  a  fleet  experimentation  exercise.”  This 
specification  would  arguably  place  considerable  contractor  emphasis  on  improving 
knowledge  flow  and  work  performance  of  users  at  the  tactical  edges  of  the  warfare 
organizations  targeted  for  the  acquisition  and  implementation  of  their  IS.  It  would  also 
appear  likely  to  help  bridge  the  gap  between  acquisition  efficiency  and  warfare  efficacy. 

Building  on  these  results,  one  can  now  apply  the  approach  described  in  this  report  to 
any  number  of  IS  acquisitions  and  use  end  customer  performance  as  an  objective  measure 
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of  IS  efficacy.  This  application  will  require  some  venue  for  (at  least  partially  controlled) 
experimentation  (e.g.,  in  the  laboratory,  via  field  experiments,  phased  or  blocked 
implementation),  but  the  potential  benefit  is  huge.  Moreover,  in  addition  to  using  dynamic 
knowledge  and  process  measurement  as  illustrated  here  for  evaluation,  one  can  leverage 
the  same  set  of  measures  to  specify  IS  in  the  conceptualization,  design,  and  development 
phases.  Essentially,  end  customer  performance  becomes  an  objective  design  consideration 
through  this  revolutionary  approach. 

In  terms  of  measuring  AWF  quality,  this  research  establishes  stronger  and  more 
direct  linkages  between  what  acquisition  personnel  know  (especially  focused  internally  on 
acquisition  organizations  and  processes)  and  what  warriors  on  the  tactical  edges  of 
organizations  need  (especially  IS  that  improve  warfare  efficacy),  and  it  provides  a  set  of 
dynamic  knowledge  and  performance  measures  that  can  be  used  to  bridge  the  gap  between 
acquisition  efficiency  and  warfare  efficacy.  This  measurement  step  alone  offers  potential  to 
improve  the  effectiveness  of  those  acquisition  people  and  organizations  that  implement  the 
approach  described  in  this  report;  hence,  one  new  measure  of  AWF  quality  emerges 
directly:  use  of  dynamic  knowledge  and  process  measures  to  assess  end  customer  efficacy. 
In  other  words,  the  working  hypothesis  is  that  those  acquisition  people  and  organizations 
that  use  this  approach  will  be  more  effective  than  those  that  do  not;  hence,  simply  assessing 
the  extent  to  which  this  approach  is  used  may  become  an  important,  complementary 
measure  of  AWF  quality. 

Further,  results  from  this  research  suggest  that  personnel  in  the  AWF  may  benefit 
from  increased  understanding  of  the  end  customers  for  whom  they  acquire  information  and 
other  systems.  The  acquisition  system  as  a  whole  provides  program  offices,  liaisons,  needs 
determination  and  justification  steps,  milestone  and  oversight  authorities,  operational  testing 
and  evaluation,  and  myriad  other  steps  seeking  to  represent  end  customers.  Nonetheless, 
there  may  be  no  substitute  for  acquisition  personnel  who  understand  their  customers  in 
considerable  detail. 

These  results  do  not  suggest  that  procurement  clerks  should  be  outfitted  with 
helmets,  rifles,  and  boots  and  then  sent  to  the  tactical  edges  of  warfare  organizations,  or 
that  warriors  on  such  tactical  edges  should  be  given  procurement  assignments;  rather,  it 
suggests  that  by  examining  the  key  warfare  processes  performed  at  the  tactical  edges — and 
in  particular,  understanding  the  most  important  dynamic  knowledge  and  performance 
characteristics  of  such  processes — even  procurement  clerks  in  offices  half  a  world  away 
may  gain  important  insight  into  their  end  customers — insight  that  may  lead  to  improved 
workforce  quality  and  that  can  be  measured. 
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